ci(e2e): report Cypress retries in CI so flaky specs are visible by fra-shipper · Pull Request #3033 · Chainlit/chainlit

fra-shipper · 2026-08-31T05:04:17Z

Root cause

cypress.config.ts sets retries: 3, so a spec that fails and then passes on retry reports the job green with no trace anywhere a human will look. #3023 is a concrete example: a genuine oauth_auth failure on windows-latest-3 was absorbed by a retry, and finding it required pulling and grepping job logs by hand. The existing after:spec handler discarded both of its arguments and only called killChainlit(), so this information was never captured.

Fix

Implements Option B from the issue (self-contained, no Cypress Cloud dependency, works on forked PRs):

cypress/support/retryReport.ts adds collectRetriedTests, a pure function that flags a test as retried when its overall state is passed but at least one attempt has state === 'failed'. Tests that exhaust every retry and still fail are excluded, since those are already visible via the job's exit status.
cypress.config.ts's after:spec handler now receives (spec, results), accumulates retried tests for the shard, and writes them to cypress/reports/retries.json after every spec (so a killed job still leaves partial data).
.github/workflows/e2e-tests.yaml uploads that file as a per-shard artifact (cypress-retries-<os>-<containers>, same pattern as the existing screenshot upload), and a new report-retries job downloads every shard's artifact, merges them, and writes a table to $GITHUB_STEP_SUMMARY.
When retries are found on a same-repo PR, the job also posts (or updates) a single PR comment. Top-level permissions: read-all is unchanged; the report-retries job itself is granted pull-requests: write, and .github/workflows/ci.yaml's e2e-tests: call site now also grants contents: read / pull-requests: write on the reusable-workflow job entry — a reusable workflow can only narrow permissions inherited from its caller, never widen them, so without this the grant inside e2e-tests.yaml would have had no effect even on same-repo PRs. On forked PRs GITHUB_TOKEN is read-only, so the comment step is skipped ahead of time based on pull_request.head.repo.full_name and additionally wrapped in try/catch as a fallback; it never uses pull_request_target. The step summary is always the primary, unconditional output.

The unrelated dead read-only-banner testid assertion in cypress/e2e/thread_resume/spec.cy.ts flagged in the issue as a separate item is left untouched, as the issue itself scopes it out.

Testing

node --test cypress/support/retryReport.test.ts -> 3/3 pass (a test passing on first attempt is ignored, a test that fails then passes is reported with its attempt count, a test that fails every attempt is ignored). Verified this is a genuine regression test: temporarily reverted collectRetriedTests to a no-op returning [] (matching the old discard-everything after:spec behavior) and reran — the retry-detection case failed with an AssertionError as expected — then restored the real implementation and confirmed 3/3 pass again.
npx prettier --check on all touched files -> clean.
npx eslint on cypress.config.ts, cypress/support/retryReport.ts, cypress/support/retryReport.test.ts -> clean.
npx tsc --noEmit -p tsconfig.json -> no new errors from this change (only 4 pre-existing TS2428 WeakMap errors from Cypress's vendored lodash types, confirmed present identically on a clean checkout via git stash -u, and unrelated to this change — this tsconfig is also not wired into any CI script; pnpm type-check only recurses into the frontend/libs workspace packages).
actionlint on the modified workflow files -> clean, exit 0.
The modified e2e-tests.yaml was parsed with yaml.safe_load to confirm it still parses correctly and the job list is ['prepare', 'e2e-tests', 'report-retries'] as intended.
The permissions: fix in ci.yaml (a reusable-workflow call site can only narrow, never widen, the caller's inherited permissions) was verified by re-reading the affected files against GitHub's documented permission-inheritance semantics for workflow_call, since that runtime behavior cannot be exercised locally or with actionlint (which validates syntax, not inheritance semantics).
Husky's pre-commit hook (lint-staged, which runs format:files, lint:fix, and actionlint on staged files) ran on every commit and made no further modifications, confirming the diff was already compliant.
Not run: pnpm test:e2e (the full Cypress suite), since it requires the full Chainlit backend and a live browser; the pure retry-detection logic is covered by the node:test regression test instead, and the artifact-upload/aggregation job will be exercised directly by this repo's own e2e-tests CI workflow on this PR.

Summary by cubic

Makes Cypress retries visible in CI so specs that fail once and pass on retry no longer produce a silently green run (#3024). The workflow now records and reports those tests without changing the pass/fail result; tests that still fail remain governed by the existing job failure.

Writes per-shard cypress/reports/retries.json, uploads it, and merges available reports into $GITHUB_STEP_SUMMARY.
Creates or updates one marker PR comment on same-repo PRs, including clearing it when a later run has no retries; forked PRs use the summary because their token is read-only.
Grants pull-requests: write through the reusable workflow call and adds the retry-report unit tests to CI via pnpm test:unit.
Guards the report hook when Cypress runs interactively, where results are unavailable.

^{Written for commit 209adbf. Summary will update on new commits.}

cypress.config.ts sets retries: 3, so a spec that fails and then passes on retry reports the job green with no trace anywhere a human will look (Chainlit#3023 was a real oauth_auth failure absorbed this way). Extend the after:spec handler, which previously discarded both of its arguments, to record tests where an attempt failed but the spec still passed, and write them to cypress/reports/retries.json per shard. Each e2e-tests.yaml matrix job now uploads that file as an artifact, and a new report-retries job downloads every shard's artifact, merges them, and writes a table to $GITHUB_STEP_SUMMARY. When a PR has retries, the job also posts or updates a best-effort PR comment; on forked PRs GITHUB_TOKEN is read-only, so the comment is skipped and the step summary remains the only, always-available output. Tested with cypress/support/retryReport.test.ts (node --test), which fails against the previous discard-everything behavior and passes against the new detection logic.

ci.yaml's e2e-tests: job called e2e-tests.yaml with no permissions override, so it inherited ci.yaml's top-level `permissions: read-all`. GitHub Actions permissions can only be downgraded through a reusable-workflow call chain, never elevated, so report-retries' own `permissions: {contents: read, pull-requests: write}` never actually took effect: the PR-comment calls were always going to 403 and get swallowed by the existing core.warning(), even on same-repo PRs. Add the matching permissions block to the caller job so the grant can flow through. Also narrow tsconfig.json's new exclude from `cypress/**/*.test.ts` to the one file that needs it, so a future cypress *.test.ts file doesn't silently lose type-checking.

cubic-dev-ai

1 issue found across 7 files

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="cypress/support/retryReport.test.ts">

<violation number="1" location="cypress/support/retryReport.test.ts:9">
P2: This regression test is never executed: nothing in any CI workflow or npm/pnpm script runs `node --test cypress/support/retryReport.test.ts`, and tsconfig.json excludes the file. The retry-detection logic therefore has no automated coverage in CI, so a future change to collectRetriedTests can silently regress. Wire the test into CI (e.g. a workflow step or a `pnpm test` script entry) so it actually runs on every PR.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

cubic-dev-ai · 2026-08-31T05:15:44Z

+const asTests = (tests: unknown) =>
+  tests as CypressCommandLine.RunResult['tests'];
+
+test('ignores a test that passed on its first attempt', () => {


P2: This regression test is never executed: nothing in any CI workflow or npm/pnpm script runs node --test cypress/support/retryReport.test.ts, and tsconfig.json excludes the file. The retry-detection logic therefore has no automated coverage in CI, so a future change to collectRetriedTests can silently regress. Wire the test into CI (e.g. a workflow step or a pnpm test script entry) so it actually runs on every PR.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At cypress/support/retryReport.test.ts, line 9: <comment>This regression test is never executed: nothing in any CI workflow or npm/pnpm script runs `node --test cypress/support/retryReport.test.ts`, and tsconfig.json excludes the file. The retry-detection logic therefore has no automated coverage in CI, so a future change to collectRetriedTests can silently regress. Wire the test into CI (e.g. a workflow step or a `pnpm test` script entry) so it actually runs on every PR.</comment> <file context> @@ -0,0 +1,63 @@ +const asTests = (tests: unknown) => + tests as CypressCommandLine.RunResult['tests']; + +test('ignores a test that passed on its first attempt', () => { + const tests = asTests([ + { </file context>

- cypress.config.ts: guard after:spec against undefined results, which cypress passes in interactive (`cypress open`) mode; the previous code threw before killChainlit() could run. - e2e-tests.yaml: reconcile the marker PR comment on every run instead of returning early when a run has zero retries, so a fixed run clears the stale "N tests retried" comment; paginate the comment lookup so the marker is found on PRs with more than one page of comments; fix "test attempt(s)" wording to "test(s)" since each retries[] entry is one test, not one attempt. - tsconfig.json: stop excluding retryReport.test.ts from type checking; enable allowImportingTsExtensions + noEmit instead so the file (which imports with an explicit .ts extension for node's native type-stripping test runner) type-checks cleanly. - package.json / tests.yaml: wire retryReport.test.ts into CI via a new test:unit script (node --test) run once from the existing "Unit & integration tests" workflow.

fra-shipper · 2026-09-03T00:23:24Z

Addressed the review findings in 209adbf:

cypress.config.ts: guarded after:spec against results being undefined, which Cypress passes in interactive mode (cypress open). Confirmed this by tracing the type definitions and reproducing the type error path; the dereference previously threw before killChainlit() ran.
.github/workflows/e2e-tests.yaml: the zero-retries early return no longer skips comment reconciliation, so a later clean run updates a stale marker comment instead of leaving it in place. Paginated the comment lookup with github.paginate so the marker is found on PRs with many comments. Fixed the wording from "N test attempt(s)" to "N test(s)" since each retries[] entry is one retried test, which can hold multiple attempts.
tsconfig.json: dropped the exclude on retryReport.test.ts and instead enabled allowImportingTsExtensions + noEmit, so the file (which imports with an explicit .ts extension for Node's native type-stripping test runner) type-checks in place. Verified with tsc --noEmit before/after.
package.json / .github/workflows/tests.yaml: added a test:unit script (node --test cypress/support/*.test.ts) and wired it into the existing "Unit & integration tests" workflow (once, not per Python-version matrix leg) so retryReport.test.ts actually runs in CI.

Verified: prettier --check, eslint, tsc --noEmit on the touched TS files, pnpm --filter @chainlit/app type-check, and node --test cypress/support/retryReport.test.ts (3/3 passing) all green locally.

cubic-dev-ai

1 issue found across 5 files (changes from recent commits).

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".github/workflows/e2e-tests.yaml">

<violation number="1" location=".github/workflows/e2e-tests.yaml:155">
P2: When a shard is killed before `after:spec`, or retry-artifact download fails, this branch treats the incomplete collection as a clean run and overwrites an existing PR comment with “No specs needed a retry.” Reconcile to zero retries only after confirming all expected shard reports were collected; otherwise leave the previous report unchanged and surface the collection failure.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

cubic-dev-ai · 2026-09-03T00:26:21Z

+
+            const marker = '<!-- cypress-retry-report -->';
+            const body =
+              retries.length === 0


P2: When a shard is killed before after:spec, or retry-artifact download fails, this branch treats the incomplete collection as a clean run and overwrites an existing PR comment with “No specs needed a retry.” Reconcile to zero retries only after confirming all expected shard reports were collected; otherwise leave the previous report unchanged and surface the collection failure.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At .github/workflows/e2e-tests.yaml, line 155: <comment>When a shard is killed before `after:spec`, or retry-artifact download fails, this branch treats the incomplete collection as a clean run and overwrites an existing PR comment with “No specs needed a retry.” Reconcile to zero retries only after confirming all expected shard reports were collected; otherwise leave the previous report unchanged and surface the collection failure.</comment> <file context> @@ -152,26 +151,34 @@ jobs: const marker = ''; - const body = `${marker}\n**${retries.length} Cypress test attempt(s) were retried in this run.** See the job summary for details.`; + const body = + retries.length === 0 + ? `${marker}\n**No specs needed a retry in this run.**` + : `${marker}\n**${retries.length} Cypress test(s) were retried in this run.** See the job summary for details.`; </file context>

fra-shipper added 2 commits August 30, 2026 18:13

fra-shipper requested review from asvishnyakov, hayescode and sandangel as code owners August 31, 2026 05:04

cubic-dev-ai Bot reviewed Aug 31, 2026

View reviewed changes

cubic-dev-ai Bot reviewed Sep 3, 2026

View reviewed changes

Navigation Menu

Uh oh!

ci(e2e): report Cypress retries in CI so flaky specs are visible - #3033

ci(e2e): report Cypress retries in CI so flaky specs are visible#3033
fra-shipper wants to merge 3 commits into
Chainlit:mainfrom
fra-shipper:fix/cypress-retry-reporting

fra-shipper commented Aug 31, 2026 •

edited by cubic-dev-ai Bot

Loading

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot Aug 31, 2026

Uh oh!

Uh oh!

Uh oh!

fra-shipper commented Sep 3, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

cubic-dev-ai Bot Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

fra-shipper commented Aug 31, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Root cause

Fix

Testing

Summary by cubic

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

fra-shipper commented Sep 3, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

fra-shipper commented Aug 31, 2026 •

edited by cubic-dev-ai Bot

Loading

cubic-dev-ai Bot left a comment •

edited

Loading